Marker generating and marker detecting system, method and program

ABSTRACT

A marker generating system is characterized in having a special feature extracting element that extracts a portion, as a special feature, including a distinctive pattern in a video image not including a marker; a unique special feature selecting element that, based on the extracted special feature, selects a special feature of an image, as a unique special feature, that does not appear on the video image; and a marker generating element that generates a marker based on the unique special feature.

TECHNICAL FIELD

The present invention relates to marker generating and marker detectingsystem, method and program.

BACKGROUND ART

Examples of conventional video image-based object detection systems aredisclosed in Patent Documents 1-7.

The conventional video image-based object detection systems operate asfollows:

Specifically, a scheme disclosed in Patent Document 1 enables detectionof an object having any shape and estimation of its attitude bysophisticating object detection without attaching a predefined markerelaborately designed beforehand for facilitating or speeding updetection processing. The object detection scheme such as that in PatentDocument 1 employing no marker eliminates the need of a marker to beattached, whereas it poses a problem that it is relatively unreliable,has a low detection speed or is unstable. A scheme disclosed in PatentDocument 2 solves such a problem by attaching a marker having adistinctive graphic pattern that is relatively visually conspicuous toan object desired to be detected. This document argues that accuracy indetection of any predefined graphic pattern can be improved. However, itposes a problem that once a graphic pattern similar to that of thepredefined marker appears in a background by chance, these patterns areconfused with each other. Patent Documents 3 and 4 avoid this problem byempirically designing beforehand a marker of a unique shape neveranticipated to appear in a background, such as a light-emitting markerand a distinctively colored marker, respectively. A scheme disclosed inPatent Document 5 employs graphic markers of a special shape such asconcentric arcs. It also employs means of constructing a graphic markerusing an infrared reflector, for example, in combination. These schemes,however, increase cost of the marker, nevertheless still potentiallypose a problem similar to that in the method disclosed in PatentDocument 2 in a case that a pattern similar to the form of the markerappears in a background by chance or the marker is placed incircumstances, such as outdoors, containing disturbance that may impairuniqueness of the form of the marker. Moreover, the marker must beempirically designed by a method requiring skill to have a form probablyunaffected by a background or turbulence, or designed by atrial-and-error process in an actual operating environment. Furthermore,since the marker may be observed with geometric distortion due to thedegree of freedom in positioning a camera in imaging, it is necessary toprepare a detection scheme taking account of such distortion. Inaddition to an increase of computational cost for detection takingaccount of geometrical distortion, it is likely to increase apossibility that the distorted marker becomes similar to a backgroundpattern by chance. The schemes disclosed in Patent Documents 6 and 7involve a highly complicated marker of a graphic pattern that neverbecomes similar to the background pattern by chance. They may embed aredundant code in a marker itself for verifying identification of themarker. Whereas the schemes can significantly reduce the possibility ofover-detection of a marker from a background, subtle graphicalinformation on the marker must be visualized on a video image, which maycause misdetection of the marker, so that imaging coverage must bereduced or resolution of an imaging device must be increased, resultingin a presumable increase of cost for implementing a decoder andreduction of the detection speed.

Patent Document 1: JP-P2000-207568A

Patent Document 2: JP-P2003-223639A

Patent Document 3: JP-P2003-256783A

Patent Document 4: JP-P2005-293141A

Patent Document 5: JP-P2005-293579A

Patent Document 6: JP-P2006-190110A

Patent Document 7: JP-P1995-254037A

DISCLOSURE OF THE INVENTION Problems to be Solved by the Invention

A first problem is low reliability in object detection of theconventional schemes. The reason thereof is that the conventionalschemes tend to over-detection when a pattern similar to a markerappears in a background, or otherwise suffer from misdetection due tofailure in reading of a marker's fine structure when complicating themarker to avoid over-detection.

A second problem is that a trial-and-error process or a skilled personis required in designing a marker. The reason thereof is that a markeris empirically designed beforehand, which causes over-detection in acase that a similar pattern is present in a practical operatingenvironment.

A third problem is an increase of cost required in disposing a marker.The reason thereof is that a reflector, a light-emitting element or thelike is employed to facilitate visual discrimination for preventing themarker from becoming similar to a background pattern.

A fourth problem is that when the positional relationship between amarker and an imaging device is unrestricted, more significant reductionof accuracy and speed in detection may result. The reason thereof isthat since design of a marker per se is based on experience and skillwithout taking account of geometrical distortion, a detection algorithmtaking account of distortion is required, and geometrical distortioncauses an increase of the frequency that the marker matches a backgroundpattern by chance.

A fifth problem is that an imaging device with higher resolution and adecoder for a more complex marker are required to reduce over-detection.The reason thereof is that the marker must be complicated.

The present invention has thus been made in view of such problems, andits object is to provide a system, method and program for markergeneration and marker detection for solving the aforementioned problems.

Another object of the present invention is to provide a system, methodand program for marker generation and marker detection enablingautomatic design of a graphic marker so that it is dissimilar to anypattern appearing in a background video image.

Still another object of the present invention is to provide a system,method and program for marker generation and marker detection enablingdetection with high accuracy and high speed for generation and detectionof a marker taking account of geometrical distortion.

Yet another object of the present invention is to provide a system,method and program for marker generation and marker detection that canbe made robust against reduction in resolution of an imaging devicebecause the marker pattern is not complicated more than necessary.

Means for Solving the Problems

The present invention for solving the aforementioned problems is amarker generating system comprising: feature extracting means forextracting as a feature a segment containing a distinctive pattern in avideo image not containing a marker; unique feature selecting means forselecting as a unique feature an image feature not appearing in saidvideo image based on said extracted feature; and marker generating meansfor generating a marker based on said unique feature.

The present invention for solving the aforementioned problems is amarker detecting system for extracting as a feature a segment containinga distinctive pattern in a video image not containing a marker,selecting as a unique feature an image feature not appearing in saidvideo image based on said extracted feature, and detecting a markergenerated based on said unique feature, said system comprising: storagemeans for storing a feature of said marker; and feature checking meansfor checking a feature in the video image subjected to detection withthe feature of said marker and notifying detection of a marker when amatch of the features is found.

The present invention for solving the aforementioned problems is amarker detecting system for extracting as a feature a segment containinga distinctive pattern in a video image not containing a marker; anddetecting a marker generated based on a unique feature selected from aportion that is not an invariant feature generated from said feature,said system comprising: storage means for storing an invariant featureof said marker; and invariant feature checking means for checking aninvariant feature in the video image subjected to detection with that ofsaid marker, and notifying detection of a marker when a match of theinvariant features is found.

The present invention for solving the aforementioned problems is amarker generating method comprising: extracting as a feature a segmentcontaining a distinctive pattern in a video image not containing amarker; selecting as a unique feature an image feature not appearing insaid video image based on said extracted feature; and generating amarker based on said unique feature.

The present invention for solving the aforementioned problems is amarker detecting method for extracting as a feature a segment containinga distinctive pattern in a video image not containing a marker,selecting as a unique feature an image feature not appearing in saidvideo image based on said extracted feature, and detecting a markergenerated based on said unique feature, said method comprising: checkinga feature in the video image subjected to detection with a feature ofsaid marker stored beforehand; and notifying detection of a marker whena match of the features is found.

The present invention for solving the aforementioned problems is amarker detecting method for extracting as a feature a segment containinga distinctive pattern in a video image not containing a marker, anddetecting a marker generated based on a unique feature selected from aportion that is not an invariant feature generated from said feature,said method comprising: checking an invariant feature in the video imagesubjected to detection with an invariant feature of said marker storedbeforehand; and notifying detection of a marker when a match of theinvariant features is found.

The present invention for solving the aforementioned problems is aprogram causing an information processing apparatus to execute theprocessing of: extracting as a feature a segment containing adistinctive pattern in a video image not containing a marker; selectingas a unique feature an image feature not appearing in said video imagebased on said extracted feature; and generating a marker based on saidunique feature.

The present invention for solving the aforementioned problems is aprogram for marker detection, for extracting as a feature a segmentcontaining a distinctive pattern in a video image not containing amarker, selecting as a unique feature an image feature not appearing insaid video image based on said extracted feature, detecting a markergenerated based on said unique feature, said program causing aninformation processing apparatus to execute the processing of: checkinga feature in the video image subjected to detection with a feature ofsaid marker stored beforehand; and notifying detection of a marker whena match of the features is found.

The present invention for solving the aforementioned problems is aprogram for marker detection, for extracting as a feature a segmentcontaining a distinctive pattern in a video image not containing amarker, and detecting a marker generated based on a unique featureselected from a portion that is not an invariant feature generated fromsaid feature, said program causing an information processing apparatusto execute the processing of: checking an invariant feature in the videoimage subjected to detection with an invariant feature of said markerstored beforehand; and notifying detection of a marker when a match ofthe invariant features is found.

The present invention for solving the aforementioned problems is amarker serving as a target of detection, which has a pattern thatmatches none of patterns in a background video image not containing amarker.

The present invention for solving the aforementioned problems is amarker serving as a target of detection, which is generated by:extracting as a feature a segment containing a distinctive pattern in avideo image not containing a marker; selecting as a unique feature animage feature not appearing in said video image based on said extractedfeature; and generating said marker based on said unique feature.

The present invention for solving the aforementioned problems is amarker serving as a target of detection, which is generated by:extracting as a feature a segment containing a distinctive pattern in abackground video image not containing a marker; acquiring an invariantfeature from said feature; selecting as a unique feature a portion thatis not said invariant feature; and generating said marker based on saidselected unique feature.

EFFECTS OF THE INVENTION

A first effect is that reliability of object detection can be improved.The reason thereof is that a marker is designed by observing abackground and avoiding similar patterns. Since the marker needs not becomplicated more than necessary, an imaging device with comparativelylow resolution may be used without failure in reading of a marker's finestructure, and misdetection is prevented.

A second effect is that there is no need for a trial-and-error processor a skilled person in designing a marker. The reason thereof is that abackground is observed and a marker is automatically designed based onthe observation.

A third effect is that cost for disposing a marker can be reduced. Thereason thereof is that since a marker can be automatically generated asa distinctive graphic pattern unlikely to be confused with a background,visual discriminability can be achieved without relying on a markermaterial such as a reflector or a light-emitting element.

A fourth effect is that even when the positional relationship between amarker and an imaging device is unrestricted, no reduction of accuracyor speed in detection is experienced. The reason thereof is that since amarker is designed by performing observation through a geometricalinvariant unaffected by geometrical distortion due to a positionalrelationship relative to an object, and designing the marker so that thegeometrical invariant is dissimilar to a background pattern, there is nopossibility that the marker matches the background pattern by chance dueto geometrical distortion, and in addition, marker detection can beachieved without any special consideration in detecting a markerdepending upon variation in positional relationship with respect to anobject by, for example, correcting geometrical distortion or making amatch taking account of distortion.

A fifth problem is that inexpensive implementation is possible withoutthe need for an imaging device with high resolution or a decoder for amore complicated marker. The reason thereof is that since the leastrequired graphical discrimination between a marker and a backgroundpattern may be sufficient, the need for complicating a marker more thannecessary is eliminated and it is not always necessary to embed aredundant code into a marker.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 A block, diagram showing a configuration of first and secondembodiments.

FIG. 2 A flow chart showing an operation of the first embodiment.

FIG. 3 A flow chart showing an operation of the second embodiment.

FIG. 4 A schematic diagram for explaining the present invention.

FIG. 5 A schematic diagram for explaining the present invention.

FIG. 6 A schematic diagram for explaining the present invention.

FIG. 7 A schematic diagram showing an operation of an example.

FIG. 8 A schematic diagram showing an operation of the example.

FIG. 9 A schematic diagram showing an operation of the example.

FIG. 10 A schematic diagram showing an operation of the example.

FIG. 11 A schematic diagram showing an operation of the example.

FIG. 12 A schematic diagram showing an operation of the example.

FIG. 13 A schematic diagram showing an operation of the example.

FIG. 14 A schematic diagram showing an operation of the example.

FIG. 15 A schematic diagram showing an operation of the example.

FIG. 16 A schematic diagram showing an operation of the example.

FIG. 17 A schematic diagram showing an operation of the example.

FIG. 18 A schematic diagram showing an operation of the example.

FIG. 19 A block diagram showing a configuration of third and fourthembodiments.

FIG. 20 A flow chart showing an operation of the third embodiment.

FIG. 21 A flow chart showing an operation of the fourth embodiment.

FIG. 22 A schematic diagram for explaining the present invention.

FIG. 23 A schematic diagram for explaining the present invention.

FIG. 24 A schematic diagram for explaining the present invention.

FIG. 25 A schematic diagram for explaining the present invention.

FIG. 26 A block diagram showing a configuration of fifth and sixthembodiments.

FIG. 27 A flow chart showing an operation of the fifth embodiment.

FIG. 28 A flow chart showing an operation of the sixth embodiment.

FIG. 29 A block diagram showing a variation of the fifth and sixthembodiments.

FIG. 30 A diagram for explaining an embodiment of the present invention.

FIG. 31 A diagram for explaining an embodiment of the present invention.

FIG. 32 A diagram for explaining an example of the present invention.

FIG. 33 A diagram for explaining the example of the present invention.

FIG. 34 A diagram for explaining the example of the present invention.

FIG. 35 A diagram for explaining the example of the present invention.

FIG. 36 A diagram for explaining the example of the present invention.

FIG. 37 A diagram for explaining the example of the present invention.

FIG. 38 A diagram for explaining the example of the present invention.

FIG. 39 A diagram for explaining the example of the present invention.

FIG. 40 A diagram for explaining the example of the present invention.

FIG. 41 A diagram for explaining the example of the present invention.

FIG. 42 A diagram for explaining the example of the present invention.

FIG. 43 A diagram for explaining the example of the present invention.

FIG. 44 A diagram for explaining another embodiment of the presentinvention.

EXPLANATION OF SYMBOLS

-   -   10, 11 Graphic marker generating section    -   20, 21 Graphic marker detecting section    -   100 Video image input means    -   101 Feature extracting means    -   102 Feature storage means    -   103 Unique feature selecting means    -   104 Marker generating means    -   105 Marker storage means    -   106 Invariant feature converting means    -   200 Feature checking means

BEST MODES FOR CARRYING OUT THE INVENTION

Embodiments of the present invention will be explained.

A embodiment of the present invention is a marker generating systemcomprising: feature extracting means for extracting as a feature asegment containing a distinctive pattern in a video image not containinga marker; unique feature selecting means for selecting as a uniquefeature an image feature not appearing in the video image based on theextracted feature; and marker generating means for generating a markerbased on the unique feature.

The embodiment of the present invention further comprises featurestorage means for calculating and storing a frequency distribution ofthe extracted features, wherein the unique feature selecting meansselects as a unique feature a portion having a frequency equal to orsmaller than a predefined value from the frequency distribution of thefeatures.

Also, the embodiment of the present invention further comprisesinvariant feature converting means for generating an invariant featurefrom the extracted feature, wherein the unique feature selecting meansselects a portion that is not the invariant feature as a unique feature.

Also, in the embodiment, the invariant feature converting meansgenerates a geometrical invariant feature from the feature.

Also, in the embodiment, the invariant feature converting meansgenerates an object-color invariant feature from the feature.

Also, in the embodiment, the invariant feature converting meansgenerates a texture invariant feature from the feature.

Also, in the embodiment, the invariant feature converting means receivesa multi-dimensional feature as an input and generates amulti-dimensional invariant feature.

Also, in the embodiment, the invariant feature converting meansgenerates an invariant feature from a combination of any one or more ofa geometrical invariant feature, an object-color invariant feature, atexture invariant feature, and a multi-dimensional invariant featurethereof.

Also, in the embodiment, the unique feature selecting means isconfigured to increase/decrease the number of unique features to beselected.

Also, in the embodiment, the unique feature selecting means calculates afrequency distribution of the invariant features, and selects as aunique feature a portion having a frequency equal to or smaller than apredefined threshold from the calculated frequency distribution of theinvariant features.

Also, in the embodiment, the unique feature selecting means controls thenumber of unique features to be selected by modifying the threshold.

Also, the embodiment of the present invention further comprises videoimage input means for inputting the video image.

Also, in the embodiment, the video image input means is capable ofpanning, tilting, zooming or moving.

Also, the embodiment of the present invention further comprises videoimage input control means for providing start and end times at which avideo image is input to the video image input means, and a time at whichimaging is performed.

Also, in the embodiment, in response to a signal from the video imageinput control means, the video image input means performs imaging,whereupon a marker is generated.

Also, in the embodiment, in response to a signal from the video imageinput control means, the video image input means performs imaging,whereupon the invariant feature converting means accumulates theinvariant feature accordingly.

Also, a embodiment of the present invention is a marker detecting systemfor extracting as a feature a segment containing a distinctive patternin a video image not containing a marker, selecting as a unique featurean image feature not appearing in the video image based on the extractedfeature, and detecting a marker generated based on the unique feature,comprising: storage means for storing a feature of the marker; andfeature checking means for checking a feature in the video imagesubjected to detection with that of the marker, and notifying detectionof a marker when a match of the features is found.

Also, the embodiment of the present invention further comprises: videoimage input means for inputting a video image subjected to detection;and video image input control means for providing start and end times atwhich a video image is input to the video image input means, and a timeat which imaging is performed, wherein the feature checking means checksa feature with a marker as the video image is input, and generates anotification when checking is succeeded at a frequency equal to orgreater than a predefined number of times.

Also, a embodiment of the present invention is a marker detecting systemfor extracting as a feature a segment containing a distinctive patternin a video image not containing a marker, and detecting a markergenerated based on a unique feature selected from a portion that is notan invariant feature generated from the feature, comprising: storagemeans for storing an invariant feature of the marker; and invariantfeature checking means for checking an invariant feature in the videoimage subjected to detection with that of the marker, and notifyingdetection of a marker when a match of the invariant features is found.

Also, in the embodiment, the invariant feature is any one of ageometrical invariant feature, an object-color invariant feature, atexture invariant feature, and a multi-dimensional invariant featurethereof, or a combination of the foregoing.

Also, in the embodiment, the invariant feature checking means notifiesdetection of a marker when one invariant feature matches.

Also, in the embodiment, the invariant feature checking means excludesfrom objects to be checked an invariant feature of the markercorresponding to that of the video image that may cause over-detection.

Also, in the embodiment, the invariant feature checking means identifiesa background portion corresponding to an invariant feature that maycause over-detection.

Also, the embodiment of the present invention further comprises meansfor confining an area in a video image subjected to detection.

Also, the embodiment of the present invention further comprises: videoimage input means for inputting a video image subjected to detection;and video image input control means for providing start and end times atwhich a video image is input to the video image input means, and a timeat which imaging is performed, wherein the invariant feature checkingmeans checks an invariant feature in the video image with that of amarker as the video image is input, and generates a notification whenchecking is succeeded at a frequency equal to or greater than apredefined number of times.

Now embodiments of the present invention will be explained in detailwith reference to the accompanying drawings.

Referring to FIG. 1, the present invention is broadly divided into agraphic marker generating section 10 and a graphic marker detectingsection 20.

First Embodiment

A first embodiment relates to the graphic marker generating section 10.The graphic marker generating section 10 is comprised of video imageinput means 100, feature extracting means 101, feature storage means102, unique feature selecting means 103, and marker generating means104.

The graphic marker generating section 10 is for, before actuallyattaching a marker to an object and performing marker detection, in anenvironment in which marker detection is applied, observing a backgroundpattern, that is, a scene excluding the object to be detected, obtainingan occurrence frequency distribution of features, and based on thedistribution, outputting a graphic pattern as a marker pattern that isnever present in the background pattern.

The means constituting the graphic marker generating section 10generally operate in a manner as described below.

The video image input means 100 inputs a video image containing an imageof an environment to which the present invention is applied, such as alive video image from an imaging device, a recorded video image, and adistributed video image.

The feature extracting means 101 receives a video image not containing amarker from the video image input means 100 as an input, and extractsimage features including distinctive patterns in the video image frame.

The feature storage means 102 keeps a memory area for calculating afrequency distribution of a feature output by the feature extractingmeans 101, and stores it.

The unique feature selecting means 103 selects as a unique feature aportion having a frequency of zero or smaller than a predefined value,for example, from the generated frequency distribution of the features.

The marker generating means 104 receives the unique feature as an input,and generates a marker pattern by combining it with an image patternthat can be easily detected by the feature extracting means 101, andoutputs it.

The marker storage means 105 stores therein the marker pattern output bythe marker generating means 104.

Next, an overall operation of the present embodiment will be explainedin detail with reference to FIG. 1 and the flow chart shown in FIG. 2.

First, a still-image frame that captures an environment not containing amarker, i.e., a background image, is input as a digitized frame image(Step A1 in FIG. 2).

Next, an image feature is extracted from the input still-image frame(Step A2). For the image feature, a graphically distinctive property inthe form of numeric values may be used, for example. For example, amethod disclosed by T. Tommasini, et al. in “Making good features trackbetter,” Proceedings of IEEE International Conference on Computer Visionand Pattern Recognition (1998), may be used to extract vertices of ashape of an object, intersections of linear objects, endpoints, etc.(indicated by in FIG. 5) in an image (see FIG. 4), although a series ofposition coordinate information of these points on the image may bedefined as graphical features. When applying feature extractionaccording to the method to a background scene as shown in FIG. 6,feature point groups as indicated by in FIG. 7 are obtained, andcoordinate values of these point groups can be employed as featuresthereafter. According to a method disclosed by U. Montanani in “On theoptimal detection of curves in noisy pictures,” Communications of ACM,Vol. 14 (1971), entries in an R table in which a distance from areference point and a relative angle are stored may be employed asfeatures. At that time, by defining a reference point for all featurepositions and exhaustively extracting features, marker detection may bemade robust against partial loss of features (described later). Otherexamples of feature extraction may include one using featuresrepresented by luma or chroma values themselves of pixels on an image.

All the features thus generated are output to and stored in the featurestorage means (Step A3).

Upon completion of recording of a series of features, an image featurenot appearing in the scene is selected from the stored feature groups asa unique feature (Step AA). The unique feature may be selected as afeature that does not match any background pattern, that is, a segmentof the feature space in which none of the feature groups in thebackground appears. To avoid an unexpected event in which the uniquefeature becomes similar to the background pattern afterwards due to anerror in extraction of feature points or the like, a unique feature maybe selected from a larger area of the feature space in which none of thefeatures of the background patterns is present. To implement this, sincethe implementation may be regarded as equivalent to a problem of findinga large blank space from a distribution of points in space, analgorithm, such as for example, “An Algorithm for Finding MaximalWhitespace Rectangles at Arbitrary Orientations for Document LayoutAnalysis,” Proceedings of International Conference on Document Analysisand Recognition (2003), may be used to extract a large blank space, or acenter of the resulting rectangular region that contains no featurepoint may be defined as a unique feature. Another method involvesquantizing the feature space in a mesh having a particular cell size,generating a one-dimensional or multi-dimensional histogram, anddefining the centers of mesh cells with a frequency of zero as a uniquefeature. If no mesh cell with a frequency of zero is found, the size ofthe mesh cell may be reduced to regenerate a histogram, and a uniquefeature may be selected from mesh cells with a frequency of zero ifpresent. When no mesh cell with a frequency of zero is found, thresholdprocessing may be applied to the histogram by using a predefined valueto select a unique feature from mesh cells having a value less than orequal to the predefined value.

Finally, from the unique feature extracted as described above, a markerpattern is generated (Step A5). First, a case in which vertices,intersections and endpoints in an image are employed as feature pointsas in the aforementioned example will be illustrated. When theaforementioned method disclosed in “On the optimal detection of curvesin noisy pictures” is employed, detection of feature point groupsrequired in detection of a marker pattern depends upon a feature pointdetection algorithm used in feature extraction at A2. For example,methods of generating a marker pattern are exemplified as describedbelow:

(1) a pattern having intersections positioned at the position of theunique feature;

(2) a pattern generated by repeating an operation of finding a convexhull of a unique feature and filling its inside with a specific color,and finding another convex hull again using a unique feature that is notused in the first convex hull and filling its inside with another color,until all features are selected;

(3) a pattern formed of a set of filled-in rectangles having verticeslying at the position of the unique feature and having horizontal andvertical sides; and

(4) a pattern in which nearest neighbor ones of unique feature pointsare connected by line segments.

Moreover, when luma and chroma values are employed as a feature obtainedat A2, a marker may be printed using a paint corresponding to the lumaand chroma values corresponding to the unique feature.

Furthermore, it is possible to combine the aforementioned method usingvertices, intersections and endpoints as graphical features with otherfeature information. In such a case, a marker having the brightness,color, shape corresponding to the selected unique feature may begenerated.

Next, effects of the present embodiment will be explained.

Since the present embodiment is configured to observe a background sceneand enable automatic design of a marker pattern so that it is dissimilarto the background pattern, reliability in object detection can beimproved. For a similar reason, there is no need for a trial-and-errorprocess or a skilled person in designing a marker. Since the markerneeds not be complicated more than necessary, an imaging device withcomparatively low resolution may be used without failure in reading of amarker's fine structure, and misdetection is prevented.

Moreover, since the present embodiment is configured to automaticallygenerate a marker as a distinctive pattern unlikely to be confused witha background, cost for disposing a marker can be reduced.

Finally, the present embodiment is configured so that the need forcomplicating a marker more than necessary is eliminated and it is notalways necessary to embed a redundant code into a marker because theleast required discrimination between a marker and a background patternmay be sufficient, and therefore, inexpensive implementation is possiblewithout the need for an imaging device with high resolution or a decoderfor a complicated marker.

Second Embodiment

Next, a second one of the best modes for carrying out the presentinvention will be explained in detail with reference to the accompanyingdrawings.

The graphic marker detecting section 20 of the present invention willnow be explained in detail with reference to FIG. 1. The presentinvention relates to detection of a marker pattern from a scene.

The second embodiment relates to the graphic marker detecting section20. The graphic marker detecting section 20 is comprised of video imageinput means 100, feature extracting means 101, marker storage means 105,and feature checking means 200.

These means generally operate in a manner as described below.

The video image input means 100 and feature extracting means 101 operatesimilarly to those in the first embodiment.

The marker storage means 106 stores therein marker patterns generatedbeforehand. When a marker pattern generated by the graphic markergenerating section 10 in the first embodiment is used, the markerpattern generated by the marker generating means 104 is input andstored.

The feature checking means 200 checks with a marker pattern stored inthe marker storage means 104, and notifies detection of a marker when amatch is found.

Next, an overall operation of the present embodiment will be explainedin detail with reference to FIG. 1 and the flow chart shown in FIG. 3.

First, Steps A1 and A2 in FIG. 3 operate on a video image scene fromwhich a marker is to be detected similarly to those in the firstembodiment.

Feature groups generated from the video image scene are checked with amarker pattern stored beforehand in the marker storage means 105 (StepB1). The marker pattern is converted into a feature beforehand. Forexample, in a case that one of features of the marker pattern and one offeatures generated from the video image scene have an Euclidean distanceequal to or smaller than a predefined value in the feature space, thefeatures may be regarded as a match and the accumulated number ofmatched invariant features may be defined as a score.

Moreover, when the result of checking with the marker pattern satisfiesa predefined condition, a notification that the marker pattern is foundfrom the video image scene is generated (Step B2).

When employing the aforementioned example, marker detection may beacknowledged in a case that the score exceeds a predefined value, or acondition that an accumulated value of the aforementioned Euclideandistance is equal to or smaller than a predefined value may beadditionally incorporated. As described above in the first embodiment,when determination of a unique feature is performed in a quantizedfeature space, the unique feature may be stored, and in a case that afeature from the video image scene is projected even once onto a meshcell having a frequency of zero in designing a marker, it may beregarded as contribution from the marker pattern to confirm detection,whereby marker detection can be quickly achieved. To avoidmisassociation due to noise or an error in feature extractioncalculation, the frequency of projection to be acknowledged may bedefined as a predefined value of one or more. Alternatively, when asimilar quantized feature space can be generated from a marker pattern,it may be employed. In this case, marker pattern detection may beacknowledged when an invariant feature obtained from the video imagescene matches with a feature space mesh to which the marker pattern isprojected once or a predefined number of times.

Next, effects of the best modes for carrying out the present inventionwill be explained.

Since the best modes for carrying out the present invention areconfigured to check the marker pattern with a video image scene throughan invariant, quickness and reliability in object detection can beimproved. Moreover, by simplifying a method of deciding acknowledgementor rejection of a marker pattern, it is possible to further improve thespeed of detection processing while maintaining reliability in objectdetection.

Example 1

Next, a specific example will be explained.

First, an operation of marker design will be particularly explained withreference to a case in which a graphical feature is employed.

For a video image scene as shown in FIG. 6, feature point groups asindicated by shown in FIG. 7 are generated. An exemplary operation ofselecting a unique feature from a quantized feature space will beexplained hereinbelow. FIG. 8 shows a mesh having 8×8 cells obtained bymapping feature point groups onto a feature space and quantizing theinvariant space from the result. In FIG. 8, a filled-in mesh cellindicates an occurrence frequency of non-zero, that is, indicates that aprojection of a feature point is present within the mesh cell, and anon-filled mesh cell indicates a mesh cell having an occurrencefrequency of zero.

According to the first embodiment, a unique feature is selected from thenon-filled mesh cells. An example in which a center of a non-filled meshcell is defined as a unique feature is shown in FIG. 9. In this example,a reference point, that is, an origin (0, 0), of a feature space isincluded in the unique feature. However, the origin may be excluded fromthe unique feature. Exemplary marker patterns are then generated fromthe resulting unique feature point groups according to the fourgeneration methods explained in the first embodiment, which are shown inFIGS. 10-13.

Next, an operation of marker detection will be particularly explained.The following description will be made on a procedure for quickly andstably detecting the marker shown in FIG. 13 from a video image sceneincluding the marker as shown in FIG. 14. A result of feature extractionon this scene is shown in FIG. 15. Moreover, the result is mapped ontoan invariant feature space and quantized to obtain a mesh having cellsonto which feature points are mapped, the result of which is shown incolor in FIG. 17. In a case that the marker is not present, a map of thefeature points is as shown in FIG. 16. Thus, in a case that featurepoints are present in mesh cells (bold-line boxes in FIG. 17) in whichno map should be present from the background, a decision that the markeris present may be made.

When a plurality of kinds of markers are used, or purple mesh cells inFIG. 18 may be stored corresponding to each marker pattern (thelight-colored letters “A,” “H” and “B” are examples of markers), and amarker for which most similar maps, that is, a largest number of maps,are obtained near or at the mesh cells in detection may be employed as aresult of detection.

Third Embodiment

Next, a third embodiment of the present invention will be explained indetail with reference to the accompanying drawings.

Referring to FIG. 19, the third embodiment relates to a graphic markergenerating section 11, which is similar to the graphic marker generatingsection 10 in the first embodiment except that an invariant featureconverting means 106 is included.

In the graphic marker generating section 11, the invariant featureconverting means 106 generally operates in a manner as described below.

The invariant feature converting means 106 converts a feature output bythe feature extracting means 101 into an invariant feature and outputsit to the feature storage means 102.

The other means 100, 101, 102-105 operate similarly to those in thefirst embodiment.

Next, an operation of the present embodiment will be explained in detailwith reference to FIG. 19 and the flow chart in FIG. 20.

First, as for the graphic marker generating section 11, Steps A1 and A2in FIG. 20 operate on a video image scene for which a marker isgenerated similarly to those in the first embodiment.

The resulting feature is converted into an invariant feature (Step A7).A feature output by the feature extracting means 101 is converted intoan invariant feature, which is output to the feature storage means 102.When a distinctive segment in an image is extracted and a series ofposition coordinate information on that image is defined as a graphicalfeature, conversion thereof into an invariant feature may be conductedin a manner as described below, for example. For convenience, the seriesof position coordinate information will be referred to as feature pointgroups herein. For simplification, a geometrical invariant feature in afar-away background will be explained. However, consideration should begiven so that a feature quantity is invariant even when an effect ofoptical distortion may cause shear deformative distortion in the image.It should be noted that it is easy to extend an invariant feature tothat having a higher degree of freedom in a case that a background isnot far away, which will be described later. An exemplary method ofgenerating a geometrical invariant feature from the positionalrelationship of the feature point groups will now be explained, whereinthe geometrical invariant feature is a feature quantity that isinvariable irrespective of variation in relative positional relationshipresulting in shear deformative distortion when a camera and a scene tobe imaged are rotated and translated relative to each other.

Three arbitrary feature points (green in FIG. 22) are selected from thefeature point groups in FIG. 5). On the other hand, an invariant featurespace is defined as a two-dimensional plane spanned by two orthogonalaxes (FIG. 23). One of the feature points selected from the featurepoint groups is associated with an origin in the invariant featurespace. Two other points are associated with position coordinates (1, 0)and (0, 1), respectively, in the invariant feature space (green in FIG.23). These three points will be referred to as bases hereinbelow. Atthat time, a one-to-one linear map from the original image space to theinvariant feature space may be defined as an affine transform. Allfeature point groups except the bases are mapped onto the invariantfeature space using the same affine transform characterized by the bases(red in FIG. 23), whereupon these feature point groups are invariantirrespective of the relative positional relationship between the cameraand scene. In practice, however, since not always the same bases can beselected from the scene, it is necessary to select bases from allpermutations and combinations of three of the feature point groups, andmap non-basis feature points with respect to each basis onto theinvariant feature space.

All the thus-created bases and maps of all feature points onto theinvariant feature space are output to and stored in the invariantfeature storage section as invariant features (Step A4). The reason whythese feature point groups are invariant against geometrical deformationis that bases selected from the marker cause the resulting invariantfeature to always match (FIG. 24) in a video image containing otherobjects (FIG. 7).

Upon completion of recording of a series of invariant feature (Step A3),Steps A4 and A5 thereafter operate similarly to those in the firstembodiment.

While the above description of the operation at Step 7 has been made ona geometrical invariant, several kinds of invariants other than thegeometrical invariant may be used. Examples of the invariants applicablein the present invention include an object-color invariant, which willbe explained hereinbelow. The color of an object may vary in imagingeven the object is the same, depending upon the color of the lightsource present in the imaging environment. If an effect of variation inlight source color can be separated out from an image, an actual objectcolor would be obtained. The resulting actual object color may be usedas an object-color invariant. A portion exhibiting specular reflectionis principally affected by the light source color and the brightnessvalue tends to saturation for the light source color component, so thatthe component may be regarded as the light source color to prevent thecolor component corresponding to the saturated portion from beingselected as an invariant feature. Besides, methods of estimating anobject color from an image that may be employed include a method byRobby T. Tan and Katsushi Ikeuchi, disclosed in “Separating ReflectionComponents of Textured Surfaces Using a Single Image,” IEEE Transactionson Pattern Analysis and Machine Intelligence, Vol. 27, No. 2, February2005, pp. 178-193; and a method by Graham D. Finlayson, Steven D.Hordley, Cheng Lu, and Mark S. Drew, disclosed in “On the Removal ofShadows from Images,” IEEE Transactions on Pattern Analysis and MachineIntelligence, Vol. 28, No. 1, January 2006, pp. 59-68.

Next, an example of a texture invariant will be explained. A brightnessdistribution for a partial region in an image is subjected to numericalcomputation, and the resulting numeric value or vector is defined as afeature quantity. Similarly to the graphical invariant, the textureinvariant is susceptible to an effect of the relative positionalrelationship between the camera and subject to be imaged, and therefore,a feature quantity insusceptible to the effect is calculated and definedas a texture invariant. For example, a feature quantity invariableagainst the distance between the camera and the object or zoom may beimplemented by converting a partial image of interest into polarcoordinates, and taking a power spectrum in the radius-vector direction.Moreover, a power spectrum may be determined again with respect to thefirst power spectrum in the azimuthal direction to obtain a featurequantity that is invariable against rotation around an optical axis ofthe camera. Besides, a method by Chi-Man Pun and Moon-Chuen Leedisclosed in “Log-Polar Wavelet Energy Signatures for Rotation and ScaleInvariant Texture Classification,” IEEE Transactions on Pattern Analysisand Machine Intelligence, Vol. 25, No. 5, May 2003, may be employed.

Furthermore, for the geometrical invariant, another kind of ageometrical invariant such as that by Richard Hartley and AndrewZisserman disclosed in “Multiple View Geometry in Computer Vision” maybe employed. When the same scene is observed by a plurality of cameras,the method disclosed in this textbook enables acquisition of informationon the distance or relative positional relationship in the depthdirection, wherein four points lying in non-identical planes may beselected as bases, and assuming that an invariant space in FIG. 23 isthree-dimensional, a three-dimensional geometric invariant may becreated. At that time, a conversion map is determined in which one offour bases selected from feature point groups is associated with anorigin of an invariant space, and feature points of the other bases areassociated with position coordinates (1, 0, 0), (0, 1, 0) and (0, 0, 1)in the invariant space, and the conversion map is used to map the otherfeatures onto the invariant space.

Moreover, two or more kinds of these and other invariants may beemployed in combination. The conversion from a feature into an invariantfeature (Step A7) and selection of a unique feature (Step A4) therefromin this case generally operate in a manner as described below.

An exemplary operation in which the aforementioned geometrical invariantand object-color invariant are employed in combination will beillustrated.

The geometrical invariant is assumed to be similar to that used in theabove description of the operation at Step A7 (FIG. 23). Theobject-color invariant employed is assumed to be the brightness value ofan object color obtained by the aforementioned method by Tan, et al. forneighboring pixels of the feature point groups extracted in determiningthe geometrical invariant. First, similarly to the aforementionedprocedure for determining a geometrical invariant, three points areselected as bases from the feature point groups, and projected onto ageometrical invariant space described in a two-dimensional plane. Anobject-color invariant corresponding to each feature position isdetermined, and a three-dimensional space including an axis orthogonalto the geometrical invariant plane, that is, an object-color invariantcoordinate, is assumed. The axes of the three-dimensional space arequantized and divided into rectangular parallelepiped mesh cells havinga predefined size, and a histogram for each rectangular parallelepipedis generated. Similar calculation is performed on all combinations ofthe bases, and values of centers of mesh cells having a histogram ofzero is defined as a unique feature (Step A4). Generation of a marker(Step A5) may be achieved by generating the marker with a position andcolor corresponding to each unique feature.

Fourth Embodiment

Next, a fourth embodiment of the present invention will be explained indetail with reference to the accompanying drawings.

Referring to FIG. 19, the embodiment of the present invention relates toa graphic marker detecting section 21, which is similar to the graphicalmarker detecting section 20 in the second embodiment except that aninvariant feature converting means 106 is included.

Now the graphic marker detecting section 21 of the present inventionwill be explained in detail with reference to FIG. 19. The presentembodiment relates to a method of detecting a marker pattern from ascene.

The present embodiment relates to the graphic marker detecting section21 and is characterized in comprising the marker detecting section 21,which is in turn characterized in comprising: video image input means100, feature extracting means 101, invariant feature converting means106, marker storage means 105, and feature checking means 200.

These means generally operate in a manner as described below.

The video image input means 100, feature extracting means 101 andinvariant feature converting means 106 operate similarly to those in theaforementioned embodiments.

The marker storage means 105 stores therein marker patterns generatedbeforehand. When a marker generated by the graphic marker generatingsection 11 is used, the marker pattern generated by the markergenerating means 104 is input and stored.

The feature checking means 200 checks with a marker pattern stored inthe marker storage means 105, and notifies detection of a marker when amatch is found.

Next, an overall operation of the present embodiment will be explainedin detail with reference to FIG. 19 and the flow chart shown in FIG. 21.

First, Steps A1, A2, A7 in FIG. 21 operate on a video image scene fromwhich a marker is to be detected similarly to those in the embodiment ofthe graphic marker generating section 11.

Invariant features generated from a video image scene are checked with amarker pattern stored beforehand in the marker storage means 105 (StepB1). The marker pattern is converted into an invariant featurebeforehand. Checking with the invariant feature generated from an actualvideo image scene is performed in the invariant feature space. Forexample, in a case that one of invariant features of the marker patternand one of invariant features generated from the video image scene havean Euclidean distance equal to or smaller than a predefined value in theinvariant feature space, the invariant features may be regarded as amatch and the accumulated number of matched invariant features may bedefined as a score.

Moreover, when the result of checking with the marker pattern satisfiesa predefined condition, a notification that the marker pattern is foundfrom the video image scene is generated (Step B2).

When employing the aforementioned example, marker detection may beacknowledged in a case that the score exceeds a predefined value, or acondition that an accumulated value of the aforementioned Euclideandistance is equal to or smaller than a predefined value may beadditionally incorporated. As described above in the first and secondembodiments, when determination of a unique feature is performed in aquantized feature space, the unique feature may be stored, and in a casethat a feature from the video image scene is projected even once onto amesh cell having a frequency of zero in designing a marker, it may beregarded as contribution from the marker pattern to confirm detection,whereby marker detection can be quickly achieved. To avoidmisassociation due to noise or an error in feature extractioncalculation, the frequency of projection to be acknowledged may bedefined as a predefined value of one or more. Alternatively, when asimilar quantized invariant feature space can be generated from a markerpattern, it may be employed. In this case, marker pattern detection maybe acknowledged when an invariant feature obtained from the video imagescene matches with an invariant space mesh to which the marker patternis projected once or a predefined number of times.

Next, effects of the third and fourth embodiments will be explained.

In addition to the effects of the first and second embodiments, thethird and fourth embodiment are configured to design a marker byperforming observation through a geometrical invariant unaffected bygeometrical distortion due to a positional relationship relative to anobject, and designing the marker so that the geometrical invariant isdissimilar to a background pattern, and therefore, there is nopossibility that the marker matches the background pattern by chance dueto geometrical distortion, and in addition, marker detection can beachieved without any special consideration in detecting a markerdepending upon variation in positional relationship with respect to anobject by, for example, correcting geometrical distortion or making amatch taking account of distortion.

Moreover, since the present embodiment is configured to design a markerby performing observation through a geometrical invariant unaffected bygeometrical distortion due to a positional relationship relative to anobject, and designing the marker so that the geometrical invariant isdissimilar to a background pattern, there is no possibility that themarker matches the background pattern by chance due to geometricaldistortion, and in addition, marker detection can be achieved withoutany special consideration in detecting a marker depending upon variationin positional relationship with respect to an object by, for example,correcting geometrical distortion or making a match taking account ofdistortion.

Furthermore, since the present embodiment is configured to automaticallygenerate a distinctive graphic pattern that is robust against variationin environment and unlikely to be confused with a background byobtaining statistic of a variety of invariants such as a geometricalinvariant and an object-color invariant or texture invariant todetermine a unique feature, detection can be stably achieved whilereducing cost for disposing a marker.

Fifth Embodiment

Next, a fifth embodiment of the present invention will be explained indetail with reference to the accompanying drawings.

A graphic marker generating section 12 in the present embodiment willnow be explained in detail with reference to FIG. 26. The presentembodiment of the invention relates to a method of generating a markerpattern from a background scene, which is similar to the graphic markergenerating section 11 in the first and third embodiments except thatvideo image input control means 107 is included.

The video image input control means 107 generally operates in a manneras described below.

The video image input control means 107 gives the video image inputmeans 100 a command to perform a video image input of a background videoimage for which a marker is to be generated at given start and end timesand at given time intervals. The command may be manually given throughthis means. Thereafter, the video image input means 100, featureextracting means 101, invariant feature converting means 106, featurestorage means 102, unique feature selecting means 103, marker generatingmeans 104, and marker storage means 105 operate similarly to those inthe aforementioned embodiments to generate one unique featuresequentially or generate a group of unique features for all images.

Next, an overall operation of the present embodiment will be explainedin detail with reference to FIG. 26 and the flow chart shown in FIG. 27.

First, Steps A2 through A8 in FIG. 27 operate on a video image scenefrom which a marker is to be detected similarly to those in the first orthird embodiment. However, input of a video image frame (Step A8) ismade continuously or intermittently in response to the command from thevideo image input control means 107. Upon input of the frame, the stepsperform processing similar to that in the first or third embodiment.Rather than outputting a unique feature on a frame-by-frame basis atStep A4, the unique feature may be stored in the invariant space, and afeature that is not observed from the background scene may be output asa unique feature as a result of observation over multiple frames.

Alternatively, as shown in the block diagram illustrated in FIG. 29, aplurality of video image input means 108-110 may be employed to input avideo image and perform a series of similar processing, or process videoimage frames in a time sequence from the plurality of video image inputmeans 108-110. The video image input means 108-110 may support amechanism allowing pan, tilt, zoom or moving. In this case, it ispossible to perform a series of similar processing of selecting a uniquefeature while making angle view control for a camera.

Next, an effect of the fifth embodiment will be explained.

Since the unique feature and marker are generated according toobservation over a long period of time or wide range or different angleviews, robustness of the invariant feature can be enhanced. By sharingan invariant space by a plurality of video image frames in common,contribution is made to reduction in computational resources required inmarker generation and detection processing. Even in a case that anobject making a non-rigid motion such as a person or an animal ispresent in the background, it is possible to generate a stable marker byobservation over a long period of time or multiple view angles.

Sixth Embodiment

Next, a sixth embodiment of the present invention will be explained indetail with reference to the accompanying drawings.

The graphic marker detecting sections 22, 23 of the present inventionwill be explained in detail with reference to FIGS. 26 and 29. Thepresent invention relates to a method of detecting a marker pattern froma video image scene, and the present embodiment of the invention issimilar to the graphic marker detecting section 11 in the first andthird embodiments, except that video image input control means 107 isincluded. The same applies to the graphic marker detecting section 23 inFIG. 29, except that a plurality of video image input means 108-110 areincluded.

The video image input control means 107 gives the video image inputmeans 100 a command to perform a video image input of a video image forwhich a marker is to be detected at given start and end times and atgiven time intervals. The command may be manually given through thismeans. Thereafter, the video image input means 100, feature extractingmeans 101, invariant feature converting means 106, marker storage means105, and marker checking means 200 operate similarly to those in theaforementioned embodiments to detect one marker sequentially or detect agroup of markers for all images.

Next, an overall operation of the present embodiment will be explainedin detail with reference to FIG. 26, FIG. 29 and the flow chart shown inFIG. 28.

First, Steps A2 through B2 in FIG. 28 operate on a video image scenefrom which a marker is to be detected similarly to those in the secondor fourth embodiment. However, input of a video image frame (Step A8) ismade continuously or intermittently in response to the command from thevideo image input control means 107. Upon input of the frame, the stepsperform processing similar to that in the first or third embodiment.Rather than generating a notification when a marker is detected from acertain frame at Step B1, circumstances of detection may be tracked, andonly a marker observed in a plurality of scenes may be output as aresult of observation over multiple frames.

Alternatively, as shown in the block diagram illustrated in FIG. 29, aplurality of video image input means 108-110 may be employed to input avideo image and perform a series of similar processing, or process videoimage frames in a time sequence from the plurality of video image inputmeans 108-110. The video image input means 108-110 may support amechanism allowing pan, tilt, zoom or moving. In this case, it ispossible to perform a series of similar processing of detecting a markerwhile making angle view control for a camera.

Next, an effect of the sixth embodiment will be explained.

Since marker detection is performed over a long period of time or widerange or different angle views, robustness of marker detection can beenhanced. By sharing the marker storage means for marker detection by aplurality of video image frames in common, contribution is made toreduction in computational resources required in marker detectionprocessing. In a case that the marker is attached to an object making anon-rigid motion such as a person or an animal or a case that an objectmaking a non-rigid motion is present in the background, it is possibleto stably detect only the marker by observation over a long period oftime or multiple view angles.

Example 2

Next, an operation of the best modes for carrying out the presentinvention will be explained with reference to a particular example.

First, an operation of designing a marker will be particularlyexplained. For a video image scene as shown in FIG. 30, feature pointgroups as indicated by circles shown in FIG. 31 are generated. Anexemplary operation of selecting a unique feature from a quantizedinvariant feature space will be explained hereinbelow. FIG. 32 shows amesh having 8×8 cells obtained by mapping feature point groups onto aninvariant space and quantizing the invariant space from the result. InFIG. 32, a non-filled mesh cell indicates an occurrence frequency ofnon-zero, that is, indicates that a projection of a feature point ispresent within the mesh cell, and a filled-in mesh cell indicates a meshcell having an occurrence frequency of zero. According to the firstembodiment, a unique feature is selected from the filled-in mesh cells.An example in which a center of a filled-in mesh cell is defined as aunique feature is shown in FIG. 33. In this example, bases in theinvariant feature space, that is, points (0, 0), (1, 0), (0, 1) areincluded in the unique feature. However, the points may be excluded.Exemplary marker patterns are then generated from the resulting uniquefeature point groups according to the four generation methods explainedin the first embodiment, which are shown in FIGS. 34-37.

Next, an operation of marker detection will be particularly explained.The following description will be made on a procedure for quickly andstably detecting the marker shown in FIG. 37 from a video image sceneincluding the marker as shown in FIG. 38. A result of feature extractionon this scene is shown in FIG. 39. Moreover, the result is mapped ontoan invariant feature space and quantized to obtain a mesh having cellsonto which feature points are mapped and filled, the result of which isshown in FIG. 41. In a case that the marker is not present, a map of thefeature points is as shown in FIG. 40. Thus, in a case that featurepoints are present in mesh cells (a hatched portion in bold-line boxesin FIG. 41) in which no map would otherwise be present from thebackground, a decision that the marker is present may be made.

When a plurality of kinds of markers are used, the circles or hatchedmesh cells in FIG. 42 may be stored corresponding to each marker pattern(“A,” “H” and “B” are examples of markers), and a marker for which mostsimilar maps, that is, a largest number of maps, are obtained near or atthe mesh cells in detection may be employed as a result of detection(FIG. 43).

Next, another embodiment of the present invention will be explained.

In this embodiment, an example in which the technique for marker anddetection according to the present invention is applied to a techniquefor detection of RFID will be explained. FIG. 44 is a diagram forexplaining this embodiment.

RFID 300 is attached thereon with a marker 301 of the present invention.The RFID 300 is detected by an RFID detecting apparatus 302, and themarker 301 is detected by a marker detection apparatus 303 providedseparately from the RFID detecting apparatus 302. It should be notedthat the marker detection apparatus 303 has a configuration similar tothat of the marker detecting section 20 as described above. A result ofdetection by the RFID detecting apparatus 302 and that by the markerdetection apparatus 303 are input to a comparing section 304 to comparethe results of detection.

When such a configuration is applied to logistics management with whicharticles passing through a predefined gate are managed, each article isattached with RFID 300 having a marker 301. The apparatus is configuredto perform both detection of the RFID 300 by the RFID detectingapparatus 302 and detection of the marker 301 by the marker detectionapparatus 303 as the article passes through the gate. Then, by comparingthe results of detection with each other at the comparing section 304,accuracy in detection of the article can be improved. While a case inwhich the marker 301 is attached to the RFID 300 has been explained inthe above example, the present invention is not limited thereto and themarker 301 and RFID 300 may be separately attached to the article, orwrapping paper for the article may be made as a marker according to thepresent invention and used as described above.

The present application claims priority based on Japanese PatentApplication No. 2007-12134 filed on Jan. 23, 2007, and Japanese PatentApplication No. 2008-3950 filed on Jan. 11, 2008, disclosures of whichare incorporated herein in its entirety.

APPLICABILITY IN INDUSTRY

The present invention may be applied to fields of video image monitoringsuch as those represented by article management and physical security.It may also be applied to fields of robot vision, mixed reality UI,content generation, and the like.

1-53. (canceled)
 54. A marker generating system comprising: featureextracting unit for extracting as a feature a segment containing adistinctive pattern in a background video image not containing a markerin an environment in which detection of a marker is performed; uniquefeature selector for selecting as a unique feature an image feature thatdoes not appear in said background video image based on said extractedfeature; and marker generator for generating a marker based on saidunique feature.
 55. A marker generating system according to claim 54,comprising feature storage unit for calculating and storing a frequencydistribution of said extracted features, wherein said unique featureselector selects as a unique feature a portion having a frequency equalto or smaller than a predefined value from said frequency distributionof the features.
 56. A marker generating system according to claim 54,comprising invariant feature converter for generating an invariantfeature that is invariant against variation of said background videoimage from said extracted feature, wherein said unique feature selectorselects a portion that is not said invariant feature as a uniquefeature.
 57. A marker generating system according to claim 56, whereinsaid invariant feature converter generates a geometrical invariantfeature from said feature.
 58. A marker generating system according toclaim 56, wherein said invariant feature converter generates anobject-color invariant feature from said feature.
 59. A markergenerating system according to claim 56, wherein said invariant featureconverter generates a texture invariant feature from said feature.
 60. Amarker generating system according to claim 56, wherein said invariantfeature converter receives a multi-dimensional feature as an input andgenerates a multi-dimensional invariant feature.
 61. A marker generatingsystem according to claim 56, wherein said invariant feature convertergenerates an invariant feature from a combination of any one or more ofa geometrical invariant feature, an object-color invariant feature, atexture invariant feature, and a multi-dimensional invariant featurethereof.
 62. A marker generating system according to claim 54, whereinsaid unique feature selector is configured to increase/decrease thenumber of unique features to be selected.
 63. A marker generating systemaccording to claim 56, wherein said unique feature selector calculates afrequency distribution of said invariant features, and selects as aunique feature a portion having a frequency equal to or smaller than apredefined threshold from the calculated frequency distribution of theinvariant features.
 64. A marker generating system according to claim63, wherein said unique feature selector controls the number of uniquefeatures to be selected by modifying said threshold.
 65. A markergenerating system according to claim 54, comprising video image inputunit for inputting said video image.
 66. A marker generating systemaccording to claim 65, wherein said video image input unit is capable ofpanning, tilting, zooming or moving.
 67. A marker generating systemaccording to claim 65, comprising video image input controller forproviding start and end times at which a video image is input to saidvideo image input unit, and a time at which imaging is performed.
 68. Amarker generating system according to claim 67, wherein, in response toa signal from said video image input controller, said video image inputunit performs imaging, whereupon a marker is generated.
 69. A markergenerating system according to claim 67, wherein, in response to asignal from said video image input controller, said video image inputunit performs imaging, whereupon said invariant feature converteraccumulates said invariant feature accordingly.
 70. A marker detectingsystem for extracting as a feature a segment containing a distinctivepattern in a background video image not containing a marker in anenvironment in which detection of a marker is performed, selecting as aunique feature an image feature that does riot appear in said backgroundvideo image based on said extracted feature, and detecting a markergenerated based on said unique feature, comprising: storage unit forstoring a feature of said marker; and feature checking unit for checkinga feature in the video image subjected to detection, which contains saidmarker in an environment in which detection of a marker is performed,with that of said marker, and notifying detection of a marker when amatch of the features is found.
 71. A marker detecting system accordingto claim 70, comprising: video image input unit for inputting a videoimage subjected to detection; and video image input controller forproviding start and end times at which a video image is input to saidvideo image input unit, and a time at which imaging is performed,wherein said feature checking unit checks a feature with a marker as thevideo image is input, and generates a notification when checking issucceeded at a frequency equal to or greater than a predefined number oftimes.
 72. A marker detecting system for extracting as a feature asegment containing a distinctive pattern in a background video image notcontaining a marker in an environment in which detection of a marker isperformed, and detecting a marker generated based on a unique featureselected from a portion that is not an invariant feature invariantagainst variation of said background video image, which was generatedfrom said feature, comprising: storage unit for storing an invariantfeature of said marker; and invariant feature checking unit for checkingan invariant feature in the video image subjected to detection, whichcontains said marker in an environment in which detection of a marker isperformed, with that of said marker, and notifying detection of a markerwhen a match of the invariant features is found.
 73. A marker detectingsystem according to claim 72, wherein said invariant feature is any oneof a geometrical invariant feature, an object-color invariant feature, atexture invariant feature, and a multi-dimensional invariant featurethereof, or a combination of the foregoing.
 74. A marker detectingsystem according to claim 72, wherein said invariant feature checkingunit notifies detection of a marker when one invariant feature matches.75. A marker detecting system according to claim 72, wherein saidinvariant feature checking unit excludes from objects to be checked aninvariant feature of said marker corresponding to that of said videoimage that may cause over-detection.
 76. A marker detecting systemaccording to claim 72, wherein said invariant feature checking unitidentifies a background portion corresponding to an invariant featurethat may cause over-detection.
 77. A marker detecting system accordingto claim 72, comprising unit for confining an area in a video imagesubjected to detection.
 78. A marker detecting system according to claim72, comprising: video image input unit for inputting a video imagesubjected to detection; and video image input controller for providingstart and end times at which a video image is input to said video imageinput unit, and a time at which imaging is performed, wherein saidinvariant feature checking unit checks an invariant feature in saidvideo image with that of a marker as the video image is input, andgenerates a notification when checking is succeeded at a frequency equalto or greater than a predefined number of times.
 79. A marker generatingmethod comprising: extracting as a feature a segment containing adistinctive pattern in a background video image not containing a markerin an environment in which detection of a marker is performed; selectingas a unique feature an image feature that does not appear in saidbackground video image based on said extracted feature; and generating amarker based on said unique feature.
 80. A marker generating methodaccording to claim 79, comprising: calculating a frequency distributionof said extracted features, and selecting as a unique feature a portionhaving a frequency equal to or smaller than a predefined value from saidfrequency distribution of the features.
 81. A marker generating methodaccording to claim 79, comprising: generating an invariant feature fromsaid extracted feature; and selecting as a unique feature a portion thatis not said invariant feature.
 82. A marker generating method accordingto claim 81, wherein said invariant feature is a geometrical invariantfeature generated from said feature.
 83. A marker generating methodaccording to claim 81, wherein said invariant feature is an object-colorinvariant feature generated from said feature.
 84. A marker generatingmethod according to claim 81, wherein said invariant feature is atexture invariant feature generated from said feature.
 85. A markergenerating method according to claim 79, comprising generating amulti-dimensional invariant feature from a multi-dimensional feature.86. A marker generating method according to claim 79, wherein saidinvariant feature is a combination of any one or more of a geometricalinvariant feature, an object-color invariant feature, a textureinvariant feature, and a multi-dimensional invariant feature thereof.87. A marker generating method according to claim 79, comprisingincreasing/decreasing the number of unique features to be selected. 88.A marker generating method according to claim 81, comprising calculatinga frequency distribution of said invariant features, and selecting as aunique feature a portion having a frequency equal to or smaller than apredefined threshold from the calculated frequency distribution of theinvariant features.
 89. A marker generating method according to claim88, comprising controlling the number of unique features to be selectedby modifying said threshold.
 90. A marker generating method according toclaim 79, comprising starting marker generation processing as imaging ofthe video image is started.
 91. A marker generating method according toclaim 79, comprising accumulating an invariant feature as imaging of thevideo image is started.
 92. A marker detecting method for extracting asa feature a segment containing a distinctive pattern in a backgroundvideo image not containing a marker in an environment in which detectionof a marker is performed, selecting as a unique feature an image featurethat does not appear in said video image based on said extractedfeature, and detecting a marker generated based on said unique feature,comprising: checking a feature in a video image subjected to detection,which contains said marker in an environment in which detection of amarker is performed, with a feature of said marker stored beforehand,and notifying detection of a marker when a match of the features isfound.
 93. A marker detecting method according to claim 92, comprisingchecking a feature in the video image subjected to detection with thatof a marker as said video image is input, and generating a notificationwhen checking is succeeded at a frequency equal to or greater than apredefined number of times.
 94. A marker detecting method for extractingas a feature a segment containing a distinctive pattern in a backgroundvideo image not containing a marker in an environment in which detectionof a marker is performed, and detecting a marker generated based on aunique feature selected from a portion that is not an invariant featureinvariant against variation of said video image, which was generatedfrom said feature, comprising: checking an invariant feature in thevideo image subjected to detection, which contains said marker in anenvironment in which detection of a marker is performed, with that ofsaid marker stored beforehand, and notifying detection of a marker whena match of the invariant features is found.
 95. A marker detectingmethod according to claim 94, wherein said invariant feature is any oneof a geometrical invariant feature, an object-color invariant feature, atexture invariant feature, and a multi-dimensional invariant featurethereof, or a combination of the foregoing.
 96. A marker detectingmethod according to claim 94, comprising notifying detection of a markerwhen one invariant feature is matched.
 97. A marker detecting methodaccording to claim 94, comprising excluding from objects to be checkedan invariant feature of said marker corresponding to that in said videoimage that may cause over-detection.
 98. A marker detecting methodaccording to claim 94, comprising identifying a background portioncorresponding to an invariant feature that may cause over-detection. 99.A marker detecting method according to claim 94, comprising confining anarea in a video image subjected to detection.
 100. A marker detectingmethod according to claim 94, comprising checking an invariant featurein said video image subjected to detection with that of a marker as thevideo image is input, and generating a notification when checking issucceeded at a frequency equal to or greater than a predefined number oftimes.
 101. A recording medium in which a program is stored, saidprogram causing an information processing apparatus to execute theprocessing of: extracting as a feature a segment containing adistinctive pattern in a background video image not containing a markerin an environment in which detection of a marker is performed; selectingas a unique feature an image feature that does not appear in saidbackground video image based on said extracted feature; and generating amarker based on said unique feature.
 102. A recording medium recordingis stored, said program for marker detection, for extracting as afeature a segment containing a distinctive pattern in a background videoimage not containing a marker in an environment in which detection of amarker is performed, selecting as a unique feature an image feature thatdoes not appear in said background video image based on said extractedfeature, and detecting a marker generated based on said unique feature,said program causing an information processing apparatus to execute theprocessing of: checking a feature in a video image subjected todetection, which contains said marker in an environment in whichdetection of a marker is performed, with that of said marker storedbeforehand, and notifying detection of a marker when a match of thefeatures is found.
 103. A recording medium recording is stored, saidprogram for marker detection, for extracting as a feature a segmentcontaining a distinctive pattern in a background video image notcontaining a marker in an environment in which detection of a marker isperformed, and detecting a marker generated based on a unique featureselected from a portion that is not an invariant feature invariantagainst variation of said video image, which was generated from saidfeature, said program causing an information processing apparatus toexecute the processing of: checking an invariant feature in the videoimage subjected to detection, which contains said marker in anenvironment in which detection of a marker is performed, with that ofsaid marker stored beforehand, and notifying detection of a marker whena match of the invariant features is found.
 104. A marker serving as atarget of detection, wherein said marker has a pattern that matches noneof patterns in a background video image not containing a marker.
 105. Amarker serving as a target of detection, wherein said marker isgenerated by extracting as a feature a segment containing a distinctivepattern in a video image not containing a marker, selecting as a uniquefeature an image feature not appearing in said video image based on saidextracted feature, and generating said marker based on said uniquefeature.
 106. A marker serving as a target of detection, wherein saidmarker is generated by extracting as a feature a segment containing adistinctive pattern in a background video image not containing a marker,acquiring an invariant feature from said feature, selecting as a uniquefeature a portion that is not said invariant feature, and generatingsaid marker based on said selected unique feature.